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Abstract 

We consider ergodic backward stochastic differential equations, in a 
setting where noise is generated by a countable state uniformly ergodic 
Markov chain. We show that for Lipschitz drivers such that a compar- 
ison theorem holds, these equations admit unique solutions. To obtain 
this result, we show by coupling and splitting techniques that uniform 
ergodicity estimates of Markov chains are robust to perturbations of the 
rate matrix, and that these perturbations correspond in a natural way to 
EBSDEs. We then consider applications of this theory to Markov decision 
problems with a risk-averse average reward criterion. 
Keywords: Ergodic BSDE, Markov Chain, Uniform Ergodicity, Nummelin 
Splitting, Risk averse control 
MSG: 60J27, 93E20, 49J55 

1 Introduction 

Much work has gone into understanding optimal control with an average cost 
criterion, over an infinite horizon (for example, see the review paper Arapos- 
tathis et al. [T], or more recent work by Guo and Hernandez-Lerma |14| . and 
references therein). Provided an underlying controlled Markov process, this cri- 
terion provides a useful method for understanding those payoffs which value the 
future as much as the present, and hence are insensitive to short-term effects. 
Much of this work is constrained to look only at costs which depend on the cur- 
rent state of the process, and at the (linear) expectation of future costs; this is, 
in this sense, a linear theory (to be precise, the corresponding Hamiltonian op- 
erators are infima taken over linear functions in the costate variable, see Section 
15. 2p . Therefore, these methods are unable to deal adequately with risk- averse 
optimization, which requires a nonlinear assessment of future costs. 

Given the known connection between BSDEs and the theory of 'nonlinear 
expectations', as defined by Peng [50] (see Coquet et al.[5] and Cohen [3] for the 
precise details of this connection), it is not unreasonable to expect that 'ergodic' 
BSDEs would provide a natural framework for understanding these nonlinear 
settings. 

In Fuhrman, Hu and Tessitore [12 (see also Richou [21 , Debussche, Hu and 
Tessitore [IH]), a class of value functions are considered which depend on the 
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average cost, not only through the current state, but on the controlled stochastic 
dynamics, and can do so in a nonlinear way. These value functions are given by 
Ergodic Backward Stochastic Differential Equations (EBSDEs), a generalisation 
of the Backward Stochastic Differential Equations developed by Pardoux and 
Peng [19] . In [12] , the stochastic dynamics are given with reference to a general 
cylindrical Wiener process in a Hilbert space, and the 'current state' is described 
by a geometrically ergodic solution to a forward stochastic differential equation. 

In this paper, we consider the ergodic BSDEs when noise is generated by 
a continuous-time discrete-state Markov chain. The basic BSDEs of this type, 
for finitely many states, were considered by one of the authors in [4] |5j |8]. 
We shall show that, with a discounted criterion, the infinite- horizon version of 
these BSDEs admits Markovian and time-invariant solutions. Assuming the 
underlying chain is uniformly ergodic, we show that discounted BSDEs preserve 
uniform ergodicity, in an appropriate sense. From here, we show that the ergodic 
BSDEs admit unique solutions. 

The paper is organized as follows. In the next section, we deal with the 
theory of discounted BSDEs. In Section |31 we show the robustness of ergodicity 
estimates of Markov chains to perturbations of the rate matrix. We give a 
novel, but natural, partial ordering of rate matrices, and show that, given the 
perturbation is not too large as determined by this ordering, any ergodicity 
estimates are transferable. This result is itself of independent interest in the 
study of ergodic properties of Markov chains. SectionSjproves the existence and 
uniqueness of bounded Markovian solutions to EBSDEs. Finally, in SectionjSl a 
numerical example and some applications of these equations in optimal ergodic 
control are discussed. We conclude with some thoughts on future extensions in 
Section |6l 

1.1 Introducing BSDEs and EBSDEs on Markov Chains 

Consider a continuous-time countable-state process X in a probability space 
(ri,J^, P), where X will be our fundamental Markov chain. Without loss of 
generality, we shall represent X as taking values from the standard basis vectors 
Ci of (where TV G N U {cxd} is the number of states, and denotes the 
space of infinite real sequences). We write X for this set of basis vectors. For 
notational simplicity, we will think of all vectors as column vectors, and denote 
by z* the transpose of z (so that z*y is the Euclidean or ^2 inner product). An 
element G £7 can be thought of as describing a path of the chain X. 

Now let {J-t} be the completion of the filtration generated by X, that is. 

Ft - a{{X,}s<t) V {B e J-oo : P(B) = 0}. 

As X is a right-continuous pure jump process which does not jump at time 0, 
this filtration is right-continuous, and we assume J- = Too = \/t<oo-^t- ^® 
shall assume that X is a Markov chain under P, in the filtration {Tt}- For 
basic theory of continuous-time countable-state Markov chains, see for example 
Rogers and Williams [22j Vol. 1, p228ff] , for the approach taken here, see Elliott, 
Aggoun and Moore [TT, Part III]). 

Let A denote the possibly infinite rate matrij|3 of the chain X. Note that 
{^t)ij > for i ^ j and J2ii^t)ij = for all j (the columns of A all sum to 0). 

^In our notation, as in A is the matrix with entries Aij, where Aij is the rate of 

jumping from state j to state i. Depending on the convention used, this is either the rate 
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We assume, for simplicity, that the entries in At are uniformly bounded, and so 
the chain is regular. 

From the Doob-Meyer decomposition (see [TTJ Appendix B]), we write our 
chain in the following way 

Xt = Xo+ [ A^X^.du + Mt, (1) 

where M is a locally- finite- variation pure-jump martingale in , and the chain 
starts in state Xq S M^. Our aim is to study EBSDEs, that is, infinite- horizon 
equations of the form 

Yt = YT + J^ [f{Xu-,Z^)^\]du~ ZldMu, 0<t<T<^, (2) 

where f : X x M.^ M is a given function, F is a real-valued cadlag stochastic 
process, Z is a predictable process in such that 

f ZldM^-.^Y. l\zurd{M)l 

Jo j JQ 

is a martingale, square integrable up to finite times, (here (•)' denotes the ith 
component of the vector), and A G M. The key points distinguishing these 
equations from 'classical' BSDEs on Markov chains (as considered in [S]) are 
that this must hold both for all t and for all T, and that the value A is a part 
of the solution. 

Our key method will be based on considering the limits of the following 
'discounted' BSDEs, 

Vr = + f[I{Xu-,ZZ) - aY^_]du - J^iZ^ydMu, 0<t<T<^, 

(3) 

the existence of solutions to which we shall first establish. 

Remark 1 . We note that the use of left limits for X and Y in the driver terms of 
(HI) , ^ initially seems unconventional, for those used to the theory of BSDEs in a 
Brownian setting. However, it is the natural approach when the driver term can 
itself jump (see [5]), it allows us to perform Girsanov transformations without 
constantly having to switch between the left and right limits of the processes, 
and as the integral is with respect to Lebcsgue measure and our processes have 
at most countably many jumps, in this case the equation is unchanged whether 
the left limits are included or not. 

Of importance will be the following process and the associated spaces. 
Definition 1. Let 

V'f := diag(^ta::) - Atdiag{x) ~ diag{x)Al, 

for X a basis vector of . Write ipt for the process i/'f"* • Then ip is a pre- 
dictable process taking values in the symmetric, positive semidefinite matrices 

matrix or its transpose. In our notation A* , the transpose of A, is the generator of the Markov 
chain. 
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in M.^^^ , with the property that 

e\( I Z*dM, 
|o,t] 



E[Z*'ipuZu]du 



for any t and any predictable processes Z of correct dimension (see JJl). For 
simplicity, we write 

and note that this is a stochastic seminorm. 
We define the following spaces of processes. 

• y G S"^ if E[sup^ Yi \ < oo and Y is cddldg, 

• Z E Hfj */-^[iio oo] W^tlWit^A °° '^^'^ ^ predictable, 

• Z E Hfj */^[ i]o t] 1 1 1 1 Ms '^■5] °° /"'^ all t < 00 and Z is predictable 
(note that this is not the usual space of processes locally in H'^, as this 
must hold for every deterministic t, rather than for a specific sequence of 
stopping times), 

• Z Z' if \\Zt — Z^Wnit = for almost all t. 

2 Discounted BSDE 

We begin with the foUowing resuh. 

Theorem 1. LetT be a finite deterministic time, and f : fix [0, T] xRxR-'^ R 
be a predictable function. If f is uniformly Lipschitz in y and z, that is, there 
exists a constant c > such that 

Ifiio, t, y, z) - f{oj, t, y', z')\ < c{\y - y'\ + \\z - z'||mJ, 
\f{oj,t,0,0)\^dt 



E 



0,T] 



< 00 



then for any ^ G L'^{J^t), there exists a unique solution (Y, Z) G S*^ x Hfj to 
the BSDE 

i = Yt- [ f{u, u, Yu, Z,,)du + f ZldM^. 

J]t,T] J\t,T] 

Proof. For the finite state case, this result is given in [3]. For the infinite state 
case, we use the martingale representation result established in [31, which natu- 
rally extends to general spaces, coupled with the existence result for BSDEs in 
general spaces estabHshed in [6]. 

We note that, unlike in [31 E], we have not assumed that J^q is trivial, that 
is, that Xq is deterministic. Hence Iq is also a random variable. This poses no 
problems for the theory of BSDEs, however it will be useful for us to note that, 
on the set {Xq = x}, we will obtain a deterministic value Yq , as J-q is nothing 
but the completion of (j{Xq). □ 
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A key result in the analysis of BSDEs is the comparison theorem. In the 
case of BSDEs with Markov Chain noise, and in general for BSDEs with jumps, 
a further condition is required to ensure that the result holds. In [71 [3] a general 
condition under which the comparison theorem holds is presented, and in [5] a 
condition specific to Markov chain BSDEs was also given. 

Definition 2. For a driver J , we say that 

• f is balanced if 

f{^,t,y,z) - f{uj,t,y,z') 



II- --'Ilk 

• f is weakly balanced if 



{z - z'YAMt > -1, 



/ f{uj,t,y,z)- f{uj,t,y,z') \ . . 

I \^^^^, AOj(z-z) AM,>-1, 



ll--^'llk 
• f is strictly balanced if 

f{^,t,y,z) - f{uj,t,y,z') 
II- --'Ilk 

for some 7 > 0, 



{z~z')*AMt > -1 + 7 



where all inequalities must hold for any y G M, any z,z' G M.^ , and up to 
indistinguishability. 

Clearly strictly balanced =4> balanced weakly balanced. Our main atten- 
tion is on those cases where / is (strictly) balanced, however, the comparison 
theorem holds assuming only that / is weakly balanced. 

Theorem 2 (Finite-time comparison theorem). Let {Y, Z) and {Y',Z') be the 
solutions to two BSDEs with drivers f and f. Suppose f is weakly balanced 
and f(LLj,t,y,z) > f'{uj,t,y, z) for all {y,z), dt x d¥-a.s. Then Yt > YJp a.s. 
implies Yt > Y/ a.s. up to indistinguishability. 

Proof. This is simply Theorem 3 of [3] , where a trivial modification of the last 
line of the proof of Lemma 1 of [3] is needed to exploit the definition of a weakly 
balanced driver. □ 

In fact, in the balanced case, the proof of the comparison theorem is easy to 
deduce from the following lemma. 

Lemma 1. If f is Lipschitz and balanced, then for any predictable processes 
Z, Z' G H^j any process Y ^ , any T < 00, the measure defined by 

^^Sf f f{^,t,Yt^,Zt)~f{u,t,Yt^,Z't) _ 

is a probability measure (where £ denotes the Doleans-Dade exponential), and 
Mt^ I {fiLo,s,Ys-,Zs)-f{oj,s,Ys^,Zi))ds+f iZ,-Z'XdMs 

J]nj.AT] J]0,tAT] 

is a Q'^ -martingale. 
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Proof. First note that 

fiLO,S,Ys-,Z,)-f{LO,S,Ys^,Zi) 
0,t/\T] \\Zs — ^sllift 

is a local martingale with quadratic variation 



[z, - z'^ydMs 



]0,tAT] II -^s ^ ^sllMt 



for c a Lipschitz constant of /. Hence we know is a true (BMO-)niartingale 
with all moments finite (see [51 Lemma 2]). Then, as / is balanced, we see that 
ALt > —1, so is a probability measure. 

To show that the desired process is a local martingale is then an application 
of Girsanov's theorem. By Holder's inequality we have, for any stopping time 
r < T, any e < 1, 

which is uniformly bounded, so AI is a true Q^-martingale. □ 

The connection between these definitions of 'balanced' drivers and the con- 
dition given in [5] is given by the following lemma, which is presented for com- 
pleteness 

Lemma 2. The following conditions are equivalent, 
(i) f is weakly balanced. 

(a) For any z,z' E M.^ , up to indistinguishability, on the set where 
ie*AtXt-)[{z - z'Yie, - Xt-)] > 

for all i, we have 

(/(l^, t, y, z) ~ /(w, t, y, z')) A > -(z - z'yAXt- a.s., 
with equality only when \\z — z'\\Mt = 0. 

Proof. Clearly the conditions are trivial on the set /(w, t, y, z) — f{uj, t, y, z') = 0, 
so we exclude this from consideration. 

(ii) (i). First note that the condition in Definition [5] is equivalent to 

[ifiLj,t,y,z)~f{LO,t,y,z'))A0]iz-z')*iXt-Xt-) > -{z-z'y^M{z^z') a.s. 

(4) 

For fixed values of i, y, z, suppose there is i with [e*AtXt-)[{z—z')* [a — 
Xt-)] < 0. Then let 

, f f{^^,t,y,z)- fiuj,t,y,z') \ 

' = [ iz-z'ne.-X,^) ^ j - 

If there is no such i, then note that {z ~ z')* AtXt- < (as it is the compensator 
of a nondecreasing process), and let 

f{uj,t,y,z)- f(uj,t,y,z') '\ 
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In either case, h*{z-z') = {f{uj, t, y, z)~ f{uj, t, y, z')) AO, l*h = and h*ei > 
for all Ci ^ Xt-. In the latter case, note also that l*{AtXt- — h) = and 
e*{AtXt- -h)>0 for all ^ Xt-. 

By considering all possible jumps, we then have that @ simplifies to 

{z-z'y[h{e,-Xt-)* +M{z-z')>Q 

for all ej with e*AtXt- > 0. This is equivalent to 

(z ~ z')*{h{e, - Xt-Y + (ej - Xt^)h* + 2i/^i](z - z') > 0. 

Ks, z — z' is at most defined up to the addition of a constant, for each e^-, we 
can suppose without loss of generality that (z — z')*ej = 0. Hence it is enough 
to show that the symmetric matrix [2tpt — hXf_ — Xt-h*] is positive definite. 
However, as this matrix is of the same form as V't, this is straightforward. 

(z) ^ (a). We know that (g]) holds. Suppose that {e*AtXt-)[{z - z')*{ei - 
^t-)] > for all i. Without loss of generality, we select a representation of z — z' 
such that (z-z')*Xt_ = and {z-z')*ej = for all j with e*AtXt- = 0. Note 
that this then implies z — z' is componentwise nonnegative. Then ^ reduces 
to 



(/(w, t, y, z) - /(cj, t, y, z')) A > - 



(z - z')*tpM{z - z') _ J2i 



Z' 



l\2 



{z-z')*{Xt) (z-z')*(Xt) 



with the convention 0/0 — oo, where = e*AtXt^. This inequality must 
hold almost surely, so it must hold in the case Xt = ej, for j maximizing 

(z — z')*ej = (z — z')j. As (z — z')i/{z — z')j < 1, we have 

ificj, t, y, z) - t, y, z')) AO "^^^ " ^'^^ 



(z - z') 



J 

> - ^ ai{z - z')i = -(z - z')*AfXt_ 

i 

as desired. □ 

Corollary 1. /// is balanced, then we also have that for any z,z' G M^, up to 
indistinguishability, if 

ie*AtXt-)[{z-z'y{e^^Xt-)] >0 

for all i, then 

f{u, t, y, z) ~ f{uj, t, y, z') > -(z - z')*AtXt- a.s., 
with equality only when \\z — z'\\m^ =0. 

Proof. The proof of (i) (ii) in Lemma [5] is sufficient to prove this result, 
where the term 'AO' is removed throughout. □ 

The following version of Tanaka's formula will be useful. We adopt the 
convention that sign(a;) = x/\x\ for x 7^ and sign(O) = 0. 
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Lemma 3. Let Y be a cadldg process of finite variation. Then the dynamics of 
\Y\ are given by 

d\Y\t^sigii{Yt.)dYt + ALl 
where ALY is the 'local time ' jump process, with 

ALj = \Yt\{l~sign{YtYt-))>0. 

Proof. The dynamics of |y| are clear except when Kj_ = or when Y jumps 
over zero, that is, when Yt- and Yt are of opposite sign. 

If Ys- = 0, then either > 0, in which case A|r^| = ALf , or = 0, 
in which case there is a version of the derivative of Y which is zero at s (as 
the measure induced by Y does not charge the point s). In either case, the 
dynamics are satisfied. 

If Yt- and Yt are of opposite sign, we have 

A|y|t = sign(rt_)Art + {A\Y\t - sign{Yt^)AYt) 

= sigu{Yt-)AYt + (|y*| - \Yt-\ - sigu{Yt-){Yt - Yt-)) 
= sign{Yt-)/^Yt + {\Yt\ ~ sign{Yt-)Yt) 
= Bign{Yt-)AYt + \Yt\{l ~ sign{Yt-Yt)) 

and so in both cases the dynamics hold. □ 

We now seek to show that there exists a bounded solution to the infinite- 
horizon BSDE with discounting. The approach used to obtain this result is based 
on Briand and Hu 2 and Royer [23]. However, in our setting, the following 
result greatly simplifies the analysis. 

Lemma 4. Let Y satisfy the dynamics 

dYt = Ptdt + Z;dMt 

for some arbitrary predictable process /?. // \Y\ < k for some k, then \e*Z\ < 2k 
for all i, for at least one representative in Hf,j That is, Z is bounded 

componentwise by twice a bound on Y . 

Proof. First note that as the jumps of M are totally inaccessible, if an inequality 
holds for every jump time, then it must hold almost everywhere on every set 
where jumps occur with positive probability, that is, up to a representative in 
H\,[ t-\oc- Now note that at any jump, 

Yt - Yt- = AYt = Z^AMt = Z;{Xt -Xt-). 

Therefore, 

\Z;{Xt- Xt-)\<2k a.s. 

We now take a representative Z such that Z*Xt- = 0, which can be done as 
Z is only ever defined up to addition of a constant. Therefore \ZtXt\ < 2k 
for every Xt which can be reached with positive probability in a single jump. 
Taking Zt = for all Xt which cannot be reached in a single jump (which can 
be done up to equality ^m), we see that \e*Z\ < 2k. □ 

Theorem 3. Let a > and f : n x R+ x M.^ ~¥ R be 
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• uniformly Lipschitz (in its third component) with respect to the \\ ■ 
norm c?P x dt-a.e. 

• balanced, in the sense of Definition\^ (omitting the y variable), and 

• such that 0)1 is uniformly bounded by C G M. 

Then there exists an adapted solution (Y.Z), with Y cddldg and Z G -^Mt-ioc 
to the equation 

Yr^Yt- [ {~aYu- + f{u;, u, Z^))du + [ Z*dM^, < t < T < oo 

J]t,T] J]t.T] 

(5) 

satisfying \Yt\ < C/a, and this solution is unique among bounded adapted solu- 
tions. 

Furthermore, if {Y'^ , Z'^) denotes the (unique) adapted solution to 

Q = Y^- f {-aYj_ + f{oj, u, Zl))du + / {ZlfdM^ (6) 

J]t,T\ J\t,T] 

then liiHT-j-oo Y^ = Yt a.s., uniformly on compact sets in t. 

Proof. Uniqueness. We first show that, if a bounded solution exists, it is 
unique. Suppose we have two bounded solutions [Y, Z) and {Y' , Z') to ([5]). For 
simphcity, write 5Y ^Y -Y' and 5Z = Z - Z'. 
For T > 0, let Qf" be the measure with density 



' {f{u:,u,Z^)~f{ u:,u,Z'^)) 

]0,-AT] P-^uIIm, 



where S denotes the Doleans-Dade exponential. As / is balanced, we can see 
that is a probability measure. By Lemma [1] it follows that 

sign{5Yu-){f{uj,u, Zu) - /(w,-", Z[^))du 

o,t] 



/ sign{SY^_)5ZldMu + ^ AL^f 
"'10, tl 



is a Qf" submartingale on [0,T]. (Note that the inclusion of sign((5yij_) simply 
exchanges Z and Z' , and does not affect the quadratic variation.) 
By Tanaka's formula and Ito's formula, we have, for all s < f < T, 

E^T[e-"'\5Yt\-e-"'\5Ys\\Fs]>Q, 

hence, 

\5Y,\ < e-"^'-'^E^T[\SYt\ < e-"(*"^)C, 

for C a bound on \SYt\. This bound is independent of T, and collapses as i — >■ oo. 
Hence \SYs\ = 0, from which we see Ys — Y^ a.s. for every s, and hence Y = Y' 
up to indistinguishability as Y and Y' are cadlag. 

Existence. We now show that a bounded solution exists. Let (Y'^ , Z"^) 
denote the solution to the time T- horizon BSDE, as defined in © . 
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First, we show that is bounded. Again, we invoke Lemma [T] and let 
denote the probabihty measure with density 

Applying Tanaka's formula and Ito's formula to e~"*|y('^|, we see that 



Y/ I < 



g{uj,u,0)\du Tt <C/a (8) 



for C a bound on |/(a;,M,0)|. Hence Y"^ is uniformly bounded. 

Second, we show that Y^ forms a Cauchy sequence in T. For any T' > T, 
we use Lemma [T] to take the probability measure 



]0,-AT] \\Zu ~ Z'^ 111^^ 

Again applying Tanaka's formula, Ito's formula and the inequality ([8]), for t < T 
we have 

\Y,^ - rn < e-"(^-*)i?Q.,.4|if 'I 1-^*] < Ce-"(^-*)/a a.s. 

Hence we see that Y^^" is a Cauchy sequence in T, and so a limit exists, and we 
denote it Yf. The desired convergence uniformly on compacts is clear, and the 
bound established in ^ also holds for Yt. 

Finally, as is uniformly bounded, so is Z'[ , by LemmaHl As Y^^" converges 
a.s. uniformly on compacts in t, its jumps converge a.s. uniformly on compacts, 
however this implies that the also converge a.s. uniformly on compacts, at 
least up to equivalence in || • \\m^. Taking Z as the limit of Z"'" , we have our 
desired solution (Y, Z) . □ 

To finish this section, we finally state a result on the existence of 'Markovian' 
solutions to these BSDEs, that is, when the BSDE solution (Y, Z) can be written 
as a function of the underlying state process X. 

Theorem 4. Suppose our Markov chain X is irreducible at every point of time. 
Consider either 

• the situation of Theorem]^ when the terminal value of the BSDE is of the 
form Q — <j){XT) for some deterministic function cj) : M.^ R, or 

• the situation of Theorem\^ 

and suppose that the driver f factors through Xt-, as a function of u, that is, 
f can be written in the form f{uj, t, ■ ■ ■) = f{Xt- (w), t, ■ ■ • ) for some function 
f. Then there exists a function v : [0,oo[ xX — > R such that Yt ~ v{t,Xt) and 
e*Zt — v(t, Ci), and this function satisfies the coupled ODE system 

^^^^ = -/(e., t, v{t, e.), v{t, •)) - e*A*Mt, ■) 
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in the setting of Theorem]^ respectively 

^^^^ - -av{t,e,) - f{e,,t,v{t, •)) - e*AMt,-), 
in the setting of Theorem\^ where v{t, •) refers to the vector in with entries 

Proof. For N finite, in the first case, this result is directly from [5]. It is easy to 
verify that the same arguments as in [5] will hold for N infinite. For the situation 
of Theorem [31 note that the finite-time approximations constructed in the 
proof of that theorem are all examples of the first case, with (p = 0. Hence 
we can find functions v'^{-, •) satisfying the desired statements, and the proven 
convergence allows us to take T — oo. The only difficulty with this is the stated 
dynamics on v{t,ei), which can easily be verified directly, as in [5]. □ 

Corollary 2. Suppose that X is time-homogeneous (so A — At is constant), 
and we are in the situation of Theorem\^ where our driver factors through Xt-, 
and does not depend on time. Then v does not depend on time, and we have 
the equation in M.^ 

av = — /(•, v) — A*v 

where v = v{t, •) is a vector in M^, and f{-,v) refers to the vector with entries 
{f{ei,v)}fLi. If f is balanced, using the natural modification of Definition \^ 
(and correspondingly Lemma\^ this equation admits a unique solution. 

Proof. Under the conditions of the corollary, as X is a Markov chain, our 
infinite- horizon BSDE ([5]) does not vary in t given the state Xt. It is then clear 
that the unique solution is a function purely of Xt . The dynamics are then the 
natural rewriting of those in Theorem 21 and the existence and uniqueness of 
the solution is the result of Theorem [31 □ 

Remark 2. It is worth noting that this is a nontrivial algebraic statement, due 
to the nonlinearity of the function /. If we knew that f{y) -\- A*v was a strictly 
monotone function, in the sense that 

(/>) - ~fW) + A*{v- v'),v - v') < -e\\v - v'f 

for some e > and all v, v' , then it would be possible to construct an existence 
result using standard techniques (see, for example [Ml p. 565]). Initially, this 
would appear to be true, at least for chains with finitely many states, using 
the fact that the nonlinear equation generates measures under which X is a 
Markov chain (see the proof of Lemma IT^ , and the Perron-Frobenius theorem 
or Krein-Rutman theorem to bound eigenvalues below zero. However, such an 
argument depends on the diagonalizability of the derivative of f{v) +A*v which 
is a non-trivial assumption. For example, one might consider the situation on 
four states with / = 0, and 



A* 



-3120 
1-320 
2-31 
2 1-3 



(9) 



which is a very well behaved Markov chain generator, however is defective as a 
matrix and so it is easy to find vectors v with v' Av > 0. Hence this argument 
fails in general. 
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3 Uniformly ergodic chains 



We aim to construct solutions to the ergodic BSDE using the solutions to the 
discounted BSDEs ([S]). As our aim is to contemplate the long-run behaviour, we 
shall now make the following two assumptions, for the remainder of the paper. 
First, the Markov chain X is time homogeneous (so At — A for all t) and 
irreducible, and second, the BSDE drivers we consider will all factor through 
X, so our BSDE solutions will be functions of the current state of the Markov 
chain, and, in particular, Corollary [5] holds. 

The key to using discounted BSDEs to approximate the ergodic BSDE is 
to study the ergodic behaviour of the Markov chain itself. We shall therefore 
make moderately restrictive assumptions on the Markov chain, sufficient to give 
explicit bounds on its convergence to its ergodic distribution, and then show 
that these bounds carry over to the solution of the relevant BSDEs. 

Definition 3. Let A4 denote the set of probability measures on X , with the 
topology inherited from considering them as a convex subset of £i{X) (the total 
variation topology, with norm ||/||tv = l/l^)!/ write PtfJ. for the law 
of Xf given Xq ^ ii. 

We say the Markov chain X is uniformly ergodic if there exists a measure 
TT on X , and constants R, p > 0, such that 

sup \\PfH - ttWtv < i?e~^* for all t. 
fi.eM 

In this case tt is the unique invariant measure for the chain. 

We now prove a few standard properties of such chains. The following lemma 
is simply a variant of [THl Theorem 16.2.2(iv)]. For simplicity, we will write Ex[-] 
for E[-\Xa = x]. 

Lemma 5. Let X be a uniformly ergodic Markov chain, and let xc be an 
arbitrary state. Let tc be the first hitting time of xc- Then for some /3 > 
(and hence for all j3 sufficiently small), 

G,^(/3) := sup£;,[e^"^] <(X). 

The following general continuity lemma allows us to take a bound, such as 
that established in Lemma [SJ and force it to converge for small /3. 

Lemma 6. Let T be a random variable, and consider G{(3) — sup^ [e*^^] , 
where v is a parameterization of measures under which the expectation is taken 
(in our case, the family of initial states of the Markov chain). If there exists 
/3* > such that G{l3*) < K , for some K < oo, then for any e > 0, 

G(/3)<l + e for all /? g [o, ^ (-^ A l) 

Note in particular that the bound does not depend on the underlying measures, 
given K and 13* . 

Proof Consider G^iP) = E^[el^'^]. For 5 > we have 
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so using the dominated convergence theorem to exchange the order of integration 
and differentiation, in the region (5 + /? < /?* we have 

Hence, fixing S — /3*/2, we have a (uniform) bound, on the derivative ■jpG^{l3) 
for ah P < /3*/2. Therefore, as ^(0) = 1, we know 

G"'(/3)<l + /3(|:G(r)) <l + /3(^i^) 

which imphes 

G''(/3)<l + e foraU /3<^(-1ai). 

Taking a supremum, we obtain the uniform bound G{(3) < 1 + e. □ 

We now return to general properties of uniformly integrable Markov chains. 

Lemma 7. Let X be a uniformly ergodic Markov chain, and let \axx\ '■— —x* Ax 
denote the rate of transitions out of state x. Then {\axxWx&x is uniformly 
hounded above 0. 

Proof. We know that the probability of being in the same state after a fixed 
timestep 5 is given by exp(— Jjoxxl)- Taking the 5 skeleton of the chain, from [ fSl 
Theorem 16.2.2(iii)] we see that the expected time until a transition from state 
X is uniformly bounded with respect to x. By a geometric distribution argument 
this expectation is given by (1 — exp(— Jjajj^:!))"^, hence \axx\ is bounded away 
from zero. □ 

Lemma 8. Let X be a uniformly ergodic Markov chain on countable states, with 
bounded transition rates. Then the discrete time chain obtained by observing X 
at each of its jump times is also uniformly ergodic. 

Proof. Let xc be an arbitrary state, which is petite as our chain has countably 
many states. Let N be the number of jumps up to hitting xc, and let the 
ith jump be denoted T^, with Tq — 0. The uniform ergodicity of a Markov 
chain is then equivalent to stating that snp^ Ex[Tn] < oo, where E^ denotes 
the expectation conditional on Xq — x. If a is an upper bound on the transition 
rates, then 



Ex[Tn]^Ex[Y,{T,-T,^^)]\ ^Ex[Y,L{t^_^^t^^E[T,-T,^^\Tt^_,] 

oo 

>Ex[j2l{T.-.<T.}^-'] ^c.''Ex[N] 

Hence sup^ Ex[N] < oo, so the Markov chain generated by observations at jump 
times is also uniformly ergodic. □ 

To prove ergodicity properties, a standard technique is based on coupling 
copies of the Markov chain (see Lindvall [16] for an overview of these methods). 
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We here shall work with the simplest form of coupling, when we take two in- 
dependent copies X and Y of our chain, which have distinct initial states, and 
we study their first meeting time. We note that throughout this section, Y will 
be used to refer to a copy of a Markov chain, rather than to the solution of a 
BSDE. 

Lemma 9. Let X and Y be two independent copies of the uniformly ergodic 
chain with rate matrix A. Let T = inf{t : Xt = Yt}. Then there exists /3 > 
such that 

G*(/3) -.^snp E,y[e^^] < oo, 

where E^y is the expectation conditional on Xq — x and Yq — y. 

Proof. Let xc be an arbitrary state, with ergodic probability n^xc)- As the 
chain is uniformly ergodic, we know 

sup \P{Yt — xc\Yo = y) — tt{xc)\ a.s t ^ oo, 
y 

so we can find t* < oo such that sup^^ \P{Yt-' = xc\Yo = y) — ■k{xc)\ < 
min{7r(a;c), 1 — 7r(a;c)}/2, in particular, P{Yt — xc) > P for all t > t* , for 
some fixed p > 0. 

Define the following sequence of stopping times. 

To = inf{i >t* ■.Xt= xc}, n = inf{t > t,_i +t* ■.Xt= xc}- 

We shall show that Tc = inf{Ti ; X^-^ = Yr^} has exponential moments, the 
result will then follow as T < Tc- As the two paths are independent, the 
definition oip implies that for each z, P(Yr- — xclJ'n^i) ^ P- Hence a geometric 
trials argument gives 

oo 

E,y[e^^^]<E,y[Y^e^^'{l-py 

oo 

= E^^.[n {e'''{l-p)E.y[e^^^^-^^-^-''^\Tr,_,+tA]_ 

i—0 j<.i 
oo 

i—0 j<.i 
oo 

= ^(e^**(l-p)G.,(/3)r 

i=Q 

^{l-{ef''il-p)G,cW))y'- 

Hence, by taking /3 small enough that e^*'G^^{l3) < (l-p)-^^^, which can be 
done by Lemma [6l we have G*{(3) < oo. □ 

Corollary 3. If X and Y are two independent copies of a uniformly ergodic 
Markov chain with bounded transition rates, and if N is the total number of 
jumps ( of both copies ) until they first meet, it follows that sup^ y E^y [e^^] < oo 
for some j3 > 0. 
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Proof. By Lemma |S1 the discrete time chain obtained by observing only when 
one of the chains jumps is also uniformly ergodic. Hence simply apply a discrete- 
time version of Lemma IHl to this setting, as the argument is identical. □ 

The following lemma provides a 'lower' exponential moment. 

Lemma 10. Let X and Y be two independent copies of a uniformly ergodic 
chain with bounded transition rates, and let N be the total number of jumps (of 
both copies) until they first meet. For any fixed 7 G ]0, 1[ we know, 

:=inf^,j7^] >0. 

Proof. By Lenima[5]the Markov chain generated by observations at jump times 
is also uniformly ergodic. Let q{x,y;n) = P{N — n\XQ — x,Yq — y). By 
Corollary [3] we know that there exists /? > such that ^„ e'^^q{x, y; n) is 
uniformly bounded for all x, y. Therefore, for some constant c, we know < 
q{x,y;n) < ce~^" for all x,y and n, hence X]^m 2^' < i-e'^ e~^"^^^. As 
we also know X^n 2/5 — li this implies that, taking m > log(i=^)//3 — 1, 
there is a positive lower bound on X]™=o^ "^hich is uniform in x and y, 

and hence q^ > 0. □ 

3.1 Perturbed rates 

We shall see later that the solutions to an infinite horizon BSDE can be viewed as 
expectations under a perturbation of the rate matrix of the underlying Markov 
chain. Hence, we wish to show that if the underlying chain is uniformly ergodic 
then this remains the case under the perturbed measure. To do so, we use a 
variant of the Nummelin splitting (see [I81 Chapter 5]), which is commonly used 
to prove ergodicity properties. 

The class of perturbations under consideration are neatly expressed through 
the following definition. 

Definition 4. Consider A and B (possibly infinite) rate matrices, that is, ma- 
trices with Aij > for i ^ j and Aij — for all j , and similarly for B . We 
shall write B > A whenever B ~ A is also a rate matrix. 

We shall say that B is controlled by A whenever there exists some 7 > 
such that J A ^ B. We shall say that B is strictly controlled by A whenever we 
also know that B — jA has diagonal entries bounded above by —7. 

Note that without loss of generality 7 can be taken to be arbitrarily small, in 
particular, we assume it is less than or equal to a positive lower bound on \axx\, 
which exists by Lemma [T] Note also that the relation >^ defined in this way is 
a valid partial ordering of rate matrices, which has not, to our knowledge, been 
previously explored. 

Theorem 5. Let A and B be rate matrices, and suppose B is strictly controlled 
by A with constant 7. //, under the measure induced by A, the process X is 
uniformly ergodic, then X is also uniformly ergodic under the measure induced 
by B. Furthermore, the constant R and the rate p of convergence to the ergodic 
distribution, in Definition\^ can be taken as functions only of A and 7, and the 
constant R can be made arbitrarily close to 1 (with a corresponding decrease in 
the rate p). 
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The proof of this theorem is the main purpose of this section. We begin by 
defining a sphtting of the Markov chain, a la (TB]. We assume the conditions 
of the theorem throughout the remainder of the section, and fix a constant 7 
satisfying Definition HI 

Definition 5. We define the split space of X to be X ^ X x {0,1}. For 

notational simplicity, we write Xq :~ X x {0} and Xi :— X x {1}, so that 
X = XoUXi. 

To split a measure v on X into a measure v on X , for y ^ X , if y is the 
projection of y, then 

v(ii) ^[^^^ i)^{y) fo'^ V £ -^0, 

\l^{y) foryeXi. 

If ij is a signed measure with a single negative component (e.g. a column of a 
rate matrix), then the negative component is not split, but is assigned to one of 
the two corresponding states in its entirety, yielding the two splittings 
Finally we define the split rate matrix B by 

fsx^"^ x^xxjOlGA-o, 
\dx^^> X = X X {1} G Xi, 

where g^ = (1 — j)^^{B — jA)x and ~ Ax. 

Intuitively, transitions occur from x E Xi following the vector Ax, and from 
X € Xq following the vector (1—7)^^(5 — 7^1), except that the result is randomly 
split between {Xq, Xi) with probabilities (1 —7, 7). Note that, as in the classical 
Nummelin splitting, this split chain has marginal transition matrix B, that is, 
for any measure q on X, we know J Bdq = J Bdq. 

Our procedure is now to consider two independent copies of our Markov 
chain on the split space, and to show that the first meeting time of these chains 
admits exponential moments. This will allow us to prove the desired uniform 
ergodicity estimates. We prove this in the following extended lemma. 

Lemma 11. Suppose B is strictly controlled by A, with constant 7 < inf^; |aa;a;|, 
and let X and Y be copies of the chain on the split space, as described in 
Definition\Si 

Let T = inf{t -.Xt^Yt} and 

H*{P) = svcpE[eP^\Xo ^x,%^ y]. 

For any e > 0, there exists a constant /3-ye such that H*{j3) < 1 + e for all 
P < /?'),£ . Furthermore, (5^^ is dependent only on 7 and A, in particular, it does 
not depend on B except through 7. 

Proof. Wc will use a renewal approach, first considering the times when a jump 
occurs that results in both X and Y being in Xi , when they weren't both in Xi 
previously. As the two chains are independent, this will occur when one chain 
is already in Xi, and the other jumps into Xi. Let Kt denote the number of 
times X or Y enters Xi up to time t, when the other is already in Xi, and tk 
denote the time of the fcth such transition. 
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We can bound H*{f3) by assuming that X and Y first meet in Xi (that is, 
we ignore any prior meetings in Xq). If we knew Kt — k, then we know that 
after tk, neither copy leaves Xi before their final meeting. As transitions out of 
Xi are independent of where in Xi the chains move, their marginal transitions 
on this set will follow the rate matrix ^A. We write E*^ for the expectation 
conditioned on not leaving Xj . Therefore, conditioning on , Yt^ and Kt — k 
and rescaling time, we see that 

E[e^^^-'^^\Xt,,Yt,,KT - fc] - E*'[eP(^^-'^'>\Xt,,K] < G*{j-'f3), 

where G*{I3) be the function defined in Lemma [TTl that is, the supremum over 
starting states of the moment generating function of the first hitting time on 
the basic (unsplit) state space X with rate matrix A. 

Write Exy for the expectation conditional on Xq = x, Iq = V-: and similarly 
Pxy for the conditional probability. We have the bound 

E,y[e^^] = E,y[E,y[e^^\Xt,,Yt,,KT] 

CO 

<E,y[j2 E[e^^\Xt, , K , Kt = k]P,y{KT = k\Xt, , F*, )" 

oo 

<J2E,y[e^*''E[e^^^-*''^\Xt,,Yt,,KT = k]P,y{KT = k\Xt,,Yt,)] 

oo 

< G*{j-'f3)Y,E,y[e^*-P,y{KT - 

oo 

< G*{^-'(3)J2iE,y{e'^''']E,y[P,y{KT = J^]) 

oo 

< G*h~'P)J2E,y[e^P*^r/'P,y{KT = kY'\ 

k=0 

(10) 

We will seek to bound the components of the above final sum. 

Probability component First consider Pxy{KT = k). Exactly as in the 
proof of Lemma [TUl from points x,y G X, let q(x, y; n) denote the probability of 
a pair of independent basic (unsplit) chains meeting in precisely n jumps, under 
the rate matrix 7 A, given starting values Xq = x,Yq = y. Then a geometric 
trials argument yields, for x ^ x x {l},y = y x {1} G Xi, 

Pfiy(i^T-0)=^<?(x,2/;n)7". 

n 

By another geometric trials argument, we see that 
Piy{KT = k)=E,y[[j2qiXt,,Yt,;n)r) U [l -^q{Xu,%;nh-)' 

n 0<i<k n 

<(l-7)(l-<?,)^ 

(11) 

where := ^^ix,yiJ2n 'ii^' V'^ n)7") > by Lemma [TUl 



17 



Expectation component Now consider Exy[e^*'']. To begin, consider 

QiP) := sup E^y[e^% 

xy 

the mgf of the first time X and Y will both be in Xi, following at least one of 
them entering Xq, maximised over starting conditions. We call a jump which 
results in one of X,Y changing between Xo and Xi a 'layer shift'. Note that 
the two chains jump at the same time with probability zero, and so the process 
which counts how many of them are in Xq is skip free. 

Now to bound Q{0), we need to bound two components - the time at 
which the first of the two chains enters Xq, and the time ti — di until both 
chains have returned to Xi . To obtain such a bound, we must also consider the 
possibility that, before ti, both chains will enter Xq. Define the stopping times 

4=mf{t>al-^ iXuYteXo}, 

a\ = inf{i >4:Xt€Xi or % e X^}, 

so that refers to the first time that precisely i of the chains are in Xq, given 
that they have both previously entered Xq precisely j times. 
Let 

Jo(/3):= sup E^y[e^^"], 

Ji(/3):= sup Eiy[e'^''i\al<h], 
xeXo,yeXi 

J2(/3):= sup E^y[el^-'^], 

x,y£Xo 

J3(/3):= sup Eiy[e^*'\h < 4], 
xeXo,yeXi 

and note that the choice of whether ± or y is in Xq or Xi is arbitrary in the 
definition of Ji and J3 . This gives Jq as the maximal mgf of the first time one 
chain enters Xq; Ji as the maximal mgf of the first time both chains are in Xq 
given one of them is now, and given this occurs before they both enter Xi; J2 
as the first time one chain enters Xi given they are both in Xq; and J3 as the 
maximal mgf of the first time both chains are in Xi given one of them is now, 
and given this occurs before they both enter Xq. 

We can now construct another geometric trials argument. If one of X, K is in 
Xq, then the next layer shift will result either in them both being in Xq or in Xi. 
The probability of both of them moving to Xi in the next layer shift depends 
on the relative rates of transition, however looking at the relative probabilities 
of transitions we find 

1 , X ^ inf{|6 

\h 

inf{|6 

XX 'y^xx 

|}7 + sup{|axx|}(l -7) 
> ■.= p 

72 + SUp{|a2;a;|}(l - 7) 

and j3 > as A is bounded. Note that p does not depend on B. Let D count 
the number of times both X and Y enter Xq before both returning to Xi. We 
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have 



(12) 



< J3(^) SUp£;s^i; 



As af is not a stopping time (due to the presence of D), we now use a 
renewal argument. 



F := sup Ex^y 



= supi;s,g E[e^^^=^ > 0]/{,5>o}e'^("i-"i) 



< sup-Ejj 
= F sup E± 



< FJ2W) suvEiJl[D>o}Ei.y[e^'-"'--^°^\D > 0] 

< FJ2{f3)MP) sup Ei^y[I^D>o}] 

x,y 

<FJ2{f3)MP){l-p). 
Hence provided J2(/3) Ji(/3)(1 — p) < 1, we have 



supi?s.g[e^(*i-"?)] < J3(/3)^^< 



J3(/3) 



1- Ji(/3)J2(/3)(l-p)' 



(13) 



By Lemma [HI if we can uniformly bound Ji and J2 , then by reducing /3 we 
have a uniform bound arbitrarily close to 1. Hence for (3 sufficiently small this 
estimate will hold, and we can combine ([T2]) and (fT3|) to give 



supEiy[e'^''] < 



Mfi)Mfi) 



x,y 



l-^i(/3)J2(/3)(l-p)' 



(14) 



Therefore, to uniformly bound the mgf of the first return time of both chains 
to Xi, it is enough to find uniform bounds on Ji for each i. 

Bounding Ji. Let denote the time of the ith jump of the chain X, and 
for j e {0,1} let 

R,{i,l3) = E[eP^^\Xt e Xj for all t S [0,r,[] =: E*^[el^^% 

so i?j(i, •) is the mgf of the ith transition of the chain, given it has not left Xj. 
Then for i > 2, 

Rj{i,[i) = E*^[e'^^^] = E*^[e^'-^E*^[e'^^'-^^-'\Xr,_,]] 
<Rj{i~l,(3)VjiP) 
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where Vj {(3) = snp^^;^_ E*^ [e^'^^ \Xo = x] is the maximum of the mgfs of the next 
transition. By definition, looking at the first jump we have Rj{l,[3) < Vj{(3), 
and so by recursion, Rj{i,(3) < (Vj(/?))'. 

To bound Jq, we can instead bound the first time that X jumps to Xq, 
ignoring any previous jumps of Y. By a geometric trials argument, we see that 

MP) =±iil- ,y-'Roi^,P) < (15) 

To bound Ji, we need only bound the time until Y enters Xq, as the con- 
dition < ^1 provides an upper bound on a^, so the conditional expectation 
is less than the unconditional expectation. Hence, by the same geometric trials 
argument, 

Mp> - 1 — n — ATA (16 

1 - (1 - -f)Vo{l3) 

To bound J2, as with Jo, we can instead bound the first time that X jumps 
to Xi, ignoring any previous jumps of Y. Hence 



J,(P)=J2{l-lh'-'Ro{^,P)<\ 



i=l 



7Vi(/3) ■ 



To bound J3, as with Ji, we see that the condition ti < (t\ provides an 
upper bound on ii, and so it is enough to bound the time until X enters Xi. 
Hence 

Therefore to find bounds on Ji for each j, we need only bound the right hand 
side of (fB ]) and ([TB| . 

As jumps of Markov chains are exponentially distributed, and conditional 
on not leaving Xi the chains move following rate matrix 7A, we can easily see 
that, as 7 < infa; \axx\, 



=sup(l 



^)"^(i-4 



As B is strictly controlled by A, and conditionally on not leaving the chains 
move following rate matrix B — 7A, we also have 

Fo(/3) = sup (1 - — ^ 1 ) < (1 - ^) 

So the right hand side of ([T5|) and ((T5)) respectively can be bounded by 



l-(l-7)Vb(/3) - 7=^-/3' l-(l-7)Vb(/3) - 7(1-7)-/?' 

which are uniformly bounded for /3 < ^(7^ A (1 — 7))- 

Therefore, for /3 sufficiently small, there is a uniform bound on ([T4|. which 
depends only on 7. Therefore, by Lemma [51 for any e > 0, there exists /? > 
such that 

g(/3) =sup£;^y[e'^*i] < 1 + e, (17) 
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and this /3 depends only on 7 and A. 

Putting the pieces together. Now it is easy to see that, with to :— 0, 

Hence, returning to (fTQ|) . we have 

snpE^yie^^] < G*{^-^P)Y,snpE^y[e^^'^]^^HnpP^y{KT = fc)i/2 

00 

< G*(7-^/3) ^(Q(2/3))'=/2(l - 7)1/2(1 _ 5^)^/2 

fc=0 

(1-7)^/' 



G*(7-V) 



1 - - 97)Q(2/3) 



This is bounded whenever (5(2/3) < (1 - g^)^^/^ and G*{j~^fi) is bounded. We 
have just shown in (ITT)) that this can be achieved with a uniform choice of /?. 
Hence, there exists a uniform choice of /3* > such that sup^^^ i?sg[e'^^] < if 
for all /?</?*. By Lemma [51 for any e > we can hence find (3^^ such that 

sup Eiy[e'^'^] < 1 + e for all ^ < fi-ye- 

To conclude, we note that this choice of /3-ye depends only on 7 and A. □ 

Corollary 4. Let X, Y be two independent copies of the Markov chain on X 
with rate matrix B. Let T be the first meeting time of these chains. Then for 
any e > 0, there exists /3 > such that sup^ y E[e^'^\XQ = x, Fo = J/] < 1 + £• 

Proof. This follows from Lemma [TTl as the chain X is simply the marginal chain 
of X ignoring the splitting, and similarly for Y . □ 

Given this result, we can now prove Theorem [5] 

Proof of Theorem\^ For any /i, £ M, consider two copies X, Y of the chain 
with rate matrix B, where Xq ^ fi, Yq ^ v. Recall we denote by Pt/i the law of 
Xt, and WfWrv = J2x \fi^)\ is a norm on M. 
Consider the coupled process 



Yt 



Yt t<T 
Xt t>T. 



It is easy to see that Yt and Yt have the same law. However, this implies, 
WPtfi ~ Pti^Wrv = J2 = ^) - ^(^* = z)\<P{T> t). 

z 

By Corollary 2] and Chernoff 's inequality, for any e > there exists /3^e > 
dependent only on 7 and A such that 

P{T >t)< E[ei^-"'^\XQ - /i,ro i^]e-~^-"^ 

< sup£;[e^^^'^|Xo = x,Yo = yje"^^'* 

< {l + e)e-^^'K 
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Therefore, 

ll-PtA^ - ^t^llw < (1 + e)e-^-*. (18) 
Replacing v with Ps~t^J■ in we see that for any s > t, 

||Pi/i - Ps^i\\TV < (1 + e)e-'^-* 

so Pt/x is a Cauchy sequence in Ai. Hence it has a hmit, Pt/i — tt, which is 
invariant under Pj. Taking v — n m psp . we then see 

llPt/i-TTllTy < (l + e)e-'^-*, 

and so the desired convergence can be guaranteed, is independent of /i, and 
occurs uniformly in B given 7 and A. □ 

4 Ergodic BSDEs 

As we now have estimates for the uniform ergodicity of our Markov chain, we 
shall use these to prove the existence of solutions to Ergodic BSDEs. The 
techniques used here are modifications of those in [TU] and [T^]. For ease of 
reference, we collect the various assumptions needed in one place. 

Assumption 1. Let X be a uniformly ergodic, time homogeneous Markov chain 
with bounded transition rates. Let f : X function satisfying the 

conditions of Theorem\^ (uniformly Lipschitz in the second component under 
II • Whit J balanced and ijj(-,0) uniformly bounded) which is also strictly balanced. 

The following lemma allows us to simply connect our analysis of uniform 
ergodicity to the theory of BSDEs, by means of Girsanov transformations. 

Lemma 12. Let Z, Z' e be any two vectors, defined up to equivalence '^m, 
and let f be our strictly balanced function. Let denote the measure under 
which X is a Markov chain with rate matrix A and initial state x. For each 
T > 0, define the measure with density 

W]0,.AT] ll^-^'llk 

Then, on the interval [0,T], X is a time-homogeneous Markov chain under 
Q^''^ , and the corresponding rate matrix B does not depend on T . Furthermore, 
B is such that the Markov chain with rate matrix B is uniformly ergodic, with 
constants R and p which do not depend on Z or Z' . 

Proof. By Lemma [1] Q^'^ is a probability measure. To show that X is a time 
homogeneous Markov chain under Q^'"^, it is enough to show that an equation 
of the form of ^ must hold, that is, for t < T we have a rate matrix B such 
that 

Xt=Xa+ / BXu-du + Q'^^'^-martingale (19) 
However, we know that for t < T, 



Xt^Xo+ I AXu-du + Mt 
]o,t] 



f{Xu-,Z)-f{Xu -,Z')^ 
0,t] J]Q,tAT] V II-^~-^'IIm„ 



Xo + I AX,,-du + I ( ' ,;^_^:||r'' V .(^ - Z') ] du + Mf 
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and so, by uniqueness of the Doob-Meyer decomposition, we can uniquely define 
a matrix B which satisfies (jl9p . through the equation 

= Ai + ^^\z^Sz%p €{Z Z') forieX 

(recalling that x ^ X is a. basis vector of R^). We know that 

VBi = l*Ai + l^^^l^J^l^^l^Vrf^-jZ -Z') = + 0, 

SO to check that i? is a valid rate matrix, we need to check that the only negative 
element can occur on the diagonal. As / is balanced, we know that for any i 
with e*Ai > 0, 

f{x,Z)-f{x,Z') I ~\\ ^ 1 

^---^ii^Z-Z) (e.-.x))>-l 

as Z and Z' are only defined up to equivalence ~m , hence at most up to the 
addition of a constant, we can assume without loss of generality that (Z — 
Z')*ei = 0, and so 

On the other hand, for this choice of representative Z — Z', we then have 
e:ru{Z - Z') = ^{e*Ai){i*{Z - Z')) 

and so, 

e:Bi = {e:Ai) (^1 - li^l^)-!^!^ (x* (Z - Z'))) > 0. 

The fact that B and A also have the same pattern of zeros can be checked in a 
similar way, as / is balanced. Therefore, X is a Markov chain under Q^'-^ with 
rate matrix B, for all T. 

As we also know / is strictly balanced, we have exactly 

nx^z)^mzi , 

Wz-zTm. iz-z))>j 

for some 7 > 0, and hence e*{B — 7^1)2; > for all e; 7^ x. Hence (as the 
columns all sum to zero) B — •jA is also a rate matrix, and hence B is controlled 
by A. We also have 

x*Bx = - ^ e*BXu- < -7 ^ e*Ax = jx*Ax, 

so 

{i*BS:) - ^{i*Ai) < ^(£M£). 

As X is uniformly ergodic under the initial measure, x*Ax is bounded away 
from zero, by Lemma [T] Therefore, for 7* < ^( '"^^ \x^Ax\ai ^^ that B is 

strictly controlled by A with constant 7*, independent of Z and Z' . 
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As we know that B is strictly controlled by A with a fixed constant 7*, we 
can apply Theorem [S] to show that there exist constants i?, p, independent of 
Z and Z', such that X is uniformly ergodic under Q^'^ with constants R and 
p. □ 

We now show that, for some sequence q;„, the solutions to the discounted 
BSDEs converge in an appropriate sense. 

Lemma 13. Let [Y^^Z") be the unique bounded adapted solution to the dis- 
counted BSDE 

Y^ = y," - / (-aF„"_ + /(X„- , ZZ))du + / {ZZYdM^, t<T<^, 

J]t,T] J]t,T] 

and let xq ^ X be an arbitrary state. By Corollary [H we know there exists a 
function : A" — > R such that Y^ — v"'{Xt). Then there exists a bound C < 00 
such that 

Iv'^ix) -v'^ixQ)\ <C", a\v'^ix)\<C" 
uniformly in x and a. Hence there exists a sequence a„ ^ such that 

(ti""(x) — u""(xo)) v{x) and q;„w""(x) — A for all x ^ X, 

for some bounded function v : X ^ M. and some A £ M. 

Proof. Let C be a bound on |/(-, 0)|. Then we see from Theorem[3]that < 
C/a. We first need to convert this into a bound which is uniform in a. To do so, 
we will view the process Y"' under a perturbed measure, and use our ergodicity 
result to control its growth. 

We first note that, by Corollary [5J the solution process Z" is constant, up 
to equivalence ^m- 

For each T > 0, define the measure with density 

/]OrAT] W^uWm., 



By Lemma [T^ under this measure X is still a uniformly ergodic Markov chain, 
with constants R, p which do not depend on a (as a only affects Z"). 

Using Girsanov's theorem in the form of Lemma [TJ we can verify that for 
any a:, a, T, 



As \v{x)\ < C/a, letting T — > cx) we obtain 

w"(a;) = lim E^..c..t\ [ e"""/(A:„_, 0)dw 



T-j-c 



0,T] 
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Therefore, for any x^x' G we have 



\v''{x)-v''{x')\ 



lim Enic.a.T 



lim En 



lim 



0,T] 



< c 



lim 

T^oo 



L J]0,T] 




/ ' 






— au 


/' /(^ 


J 


X 




\\PuS^ 



e-""/(^n-,0)d« 
e-""/(X„_,0)dM 



0,T] 



lim 

T->-oo 



< Ci? 



0,T] 



And so we have a uniform bound ~ v"(x')\ < CRp ^. In addition, we 

recaU that \av°'{x)\ < C. 

Given this, we can use a diagonal procedure to construct a sequence a„ J, 
such that 

a„w""(a;o) ^ A 

- w""(a;o) v{x) for all a; e A" 

for a function v : X ^ M. (recall that X is at most countable). Note that, for 
C" = Ci?/9-\ we have the bounds |A| < C" and < C". 

Finally, we notice that, for an arbitrary a: £ A", 

anv""{x) = a„w""(a;o) + (a;) - w""(xo)) ^ A + 

so the convergence of this sequence to A holds for all x. □ 

Theorem 6. Let v, A be as constructed in Lemma \1SI The triple {Y,Z,X), 
where 

Yt := v{Xt), e*Zt = v{ei), 
is the unique bounded Markovian solution, with v{xq) = 0, to the Ergodic BSDE 

Yt=YT+ [ [f{Xu- , Z^) -X]du+ f ZldA'U. 

J]t.T] J]t.T] 



Any other bounded solution {Y' , Z' , X') satisfies X = A', and any other bounded 
Markovian solution (Y' , Z', A') satisfies Yl ~ Yt + c for some c e R, and Z' 
Z. 

Proof. Let be the sequence constructed in Lemma[T31 As Z"" is only defined 
up to constant shifts, we know that Y^"" := t>""(At), e*Z"" :— v°'"{ei) — v°'"{xo) 
solves the discounted BSDE 

JH.T] J]t.T] 



Note that Z"" is constant in t, and by the bound established in Lemma [T51 
\e*Z"" \ is uniformly bounded. Hence, as the transition rates in A are bounded. 
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we also know ||^ is uniformly bounded, and dominated convergence yields 
lim(/Z""dM) = /(limZ"")dM. Therefore 

n 

= lim(t;""(XT)-?;""(a;o))+lim / Z"" ) ~ a„z;"" 

" " J]t,T] 

+ lim / {Z^-ydMu 

" J]t,T] 

= v{Xt)+ [ lim(/(X„_,Z"" -t>""(a;o)) -lima„w""(X,_))(iM 

J]t,T] " " 

lim(Z"" ~v°"^{xo)ydMu 
t,T] " 

= v{Xt)+ I {f{Xu-,Z)-\)du+ [ Z*dMu, 

J]t,T] J]t,T] 

that is, {v{Xt),v{-), X) is a solution to the EBSDE. 

To show the solution is unique, suppose {Y' , Z', A') is another bounded so- 
lution. Let f = y - y, Z = Z - Z' and A = A - A'. Then 

Yt = YT+ f [f{Xu- , Z) - f{Xu- , Z:) - \]du + f ZldM^ 

J]t,T] J]t,T] 

and defining the measure Qf as in ([7]) 

d^"" W]o,AT] \\Z\\Ij^ ' 

we see that 

As Y is uniformly bounded, taking T — >■ cxd we see A = 0, that is, A = A'. 
Substituting back, we see that Iq — Eqt [Yt] for all T. 

From Lemma [T^ if Y' is a bounded Markovian solution with Y/ = v'{Xt), 
we see that X is still a time homogenous uniformly ergodic Markov chain under 
QJ , on the interval [0,T]. The family of measures is then consistent, in 
that Q'^Ij^j. = Q'^ for all T < T' . Therefore, conditioning on Xq = x, we 
can extend them to a single measure under which X is a time homogenous 
uniformly ergodic Markov chain for all times, and Q^\j^t — Q'i\j^t,Xo=x for all 
T. If TT is the ergodic measure of X under Q^, we then have 

v{x) - v'{x) = lb = limEq^lYr] = / {v{y) - v'{y))dn{y), 

and we see that the right hand side is independent of x, that is, v'{x) — v{x) +c 
for some constant c e R, for all x ^ X. The equivalence ||Z — Z'||^^^ = then 
follows from Corollary [21 □ 

Corollary 5. The sequences anv"^"^ andv°'"^(x) — v°''^{xQ) constructed in Lemma 
[731 converge for any choice of sequence {an i 0}. 
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Proof. Suppose this were not the case. Then wc could construct two sequences 
with distinct hmits, both of which would yield bounded Markovian solutions to 
the EBSDE. However we have shown that there is only one such solution, which 
is a contradiction. □ 

Corollary 6. The value A in the EBSDE solution {Y,Z,X) = {v{Xt),v,X) 
satisfies 

A = / f{y,v)dTr{y), 
Jx 

and hence, if we define a signed measure /i^ on X by n^iy) = /jo oo[(^?(2/) ~ 
'K{y))dt, (which is well defined due to the uniform ergodicity of X under F) we 
have 

v{x) =c+ f{y,v)dii^{y) 
Jx 

for some c = v{x)dTr S R. 

Proof. The invariance of the ergodic distribution tt implies that, for any fixed 
time T, any g : X ^R, 

[ Er4g{XT)]dn{x) = [ Ep4g{XT-)]d7r{x) = f g{x)d-K{x). 
Jx Jx Jx 



We write 



v{Xt)+ [ {f{X^-,v)-X) 

J]0,T] 



v{x) = Ef. 
so by the stated invariance property, 

i{x)dTr{x)= / Er4v{XT)]dTr{x) + / Ei 
Jx Jx 

= / v{x)dTT{x) + 

Jx 

= / v{x)d'K{x 
Jx 



du 



+ 



t,T] 



t,T] 

Er^ [f{Xu-,v)]dTr{x) 



{f{Xu-,v) - X)du d'7r{x) 
du — TX 



f{x, v)dTr{x) 



du-TX 



= / v{x)dn{x)+T f{x,v)d'K{x)~TX 
Jx Jx 



and rearrangement yields the first result. 
Consequently, we also have 



v{x) = lim Ep. v{Xt)+ / (fiXu-,v) - X)du 



+ lim 

T-^oo 



= / v{y)dTT{y) 
Jx 

v{y)dTr{y) 

v{y)dn{y)+ / f{y,v)dtf{y). 
Jx 



]0,oo[ ^JX 



I f{y,v)dFl{y)- [ f{y,v)dn{y))du 

]0,T] "-Jx Jx ' 

f{y,v)d{PUy)-^{y)))du 



□ 
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Remark 3. If the Lipschitz constant of / were sufficiently small, when compared 
with the coefficients in the ergodicity of X, the above corollary would provide 
a simple direct proof of existence for EBSDEs. Furthermore, while we do not 
have a comparison theorem for the Y component of BSDEs, from this result, 
if c is fixed, we can differentiate v{x) in terms of f{y,v{-)), and (again given 
a small enough Lipschitz constant) solve the resultant implicit equation. From 
this analysis, we can see that, if / is sufficiently small, then v{x) is monotone 
increasing in f{x,-). 

Remark 4. This representation of v in fact provides an intuitive meaning for v. 
We see here that, up to the addition of a constant, v{x) is given by the cost /, 
integrated through time with respect to the 'deviation from ergodicity' measure 
/i^. Therefore, it is natural to think of v as giving a 'short-run additional 
expected cost' term, while A gives the long-run ergodic cost. 

We now give a variant of the comparison theorem for EBSDEs, considering 
the A component of the solution. 

Theorem 7. Let f and f be two strictly balanced drivers, and (Y, Z, A) and 
(Y',Z\X') the corresponding EBSDE solutions. Then if f{x,z) > f'{x,z) for 
allxe X, Z e R^, then A > A'. 

Proof. Take the measure with density 

dd^'^ _ f f f'{X^^,Z)-f'{X^^,Z') 

and the corresponding rate matrix B and resultant ergodic measure ttb- Then, 
if Yt = v{Xt) and F/ = v'{Xt), integrating the difference of the EBSDEs yields 

v{x) — v' {x)dnB 
En 



Yt-Y^+ {f{Xu-,Z) - f'{X^^,Z))dt - r(A - A') dTTB 

X '- "']0,T] J 

{vix) - v'{x))dTTB + [ ( [ fix, Z) - fix, Z)dnB)dt - r(A - A') 

X "']0,T] ^Jx ' 

and so 

(A-A')= / (/(.T,Z)-/'(a;,Z))d^B >0. 
J X 

□ 

Remark 5. We note that we have also shown that, given that / is strictly 
balanced and A is the generator of a uniformly ergodic chain, the vector equation 

= f{-,v)+A*v - Al 

admits a unique solution (u(-),A) £ x R, and that this solution is approxi- 
mated by the equations considered in Corollary [2] This is without any further 
reference to the monotonicity of /. This can be compared with the classical 
equation for ergodic cost, which is simply 

= f{-) + A*v-Xl. 
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Remark 6. From a numerical perspective, as we expect that /(•,«) + A*v will 
typically be 'approximately monotone' in v, solving this equation iteratively, for 
example through the equations 

Vn+1 = (A*)+(a„1 - /(•,«„)) 

where {A*)~^ denotes the Moore-Penrose pseudoinverse of A* , will often provide 
a simple numerical scheme in the finite state case. 

In the infinite state Monte-Carlo approach can be used to approx- 

imate the equations given by Corollary [SJ however the convergence of such a 
scheme lies beyond the scope of this paper. 

5 Applications and Examples 
5.1 Rate uncertainty 

As a first example, we consider the rate matrix A constructed and the driver 

f{x,v)^lj;(zQ+ min {{r-l)v*Ax}, 

re[l//3,^] 

where C ^ and /? > 1. Essentially, this driver attempts to determine the 
ergodic probability of being in the set C, however it introduces an uncertainty 
about the overall transition rate of the chain, by scaling the rates up or down 
with the parameter P, so as to minimise the probability of being in ^. A related 
BSDE over finite horizon, for a different choice of A, was considered in [51. 

For /3 = 2, we solve the EBSDE numerically using the simple algorithm 
suggested in Remark [Sj We list all possibilities for ( up to symmetries, and 
choose V such that its values in each state sum to zero. For comparison, we also 
list the ergodic probability associated with the set 



c 


v{ei) 


vie2) 


vies) 


^(64) 


A 


AC) 























{ei} 


0.1207 


-0.0172 


-0.0517 


-0.0517 


0.0345 


0.125 


{62} 


-0.0652 


0.1087 


-0.0217 


-0.0217 


0.1304 


0.375 


{61,62} 


0.1000 


0.1000 


-0.1000 


-0.1000 


0.2000 


0.500 


{61,63} 


0.1500 


-0.0500 


0.0500 


-0.1500 


0.2000 


0.500 


{61,64} 


0.0769 


-0.0769 


-0.0769 


0.0769 


0.0769 


0.250 


{62,63} 


-0.1429 


0.1429 


0.1429 


-0.1429 


0.4286 


0.750 


{61,62,63} 


0.1364 


0.1364 


0.0455 


-0.3182 


0.6364 


0.875 


{61,62,64} 


0.0294 


0.0294 


-0.1471 


0.0882 


0.2941 


0.625 


{61, 62, 63, 64} 














1 
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Note that, in the classical case without the rate uncertainty term, we would 
have A^ = 7r(C). This is clearly not the case here, due to the nonlinearity 
introduced by the rate uncertainty. In fact, we can see here that A defines, in 
some sense, an 'ergodic capacity' for the chain (however one can verify that the 
EBSDE solution is generally not given by the Choquet integral with respect to 
this capacity). Note further that while the ergodic probability does not reveal 
the asymmetry between the states of the chain, the EBSDE is affected by this 
in a nontrivial way. 
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5.2 Classical Optimal Ergodic Control 

We now give a more abstract example, indicating how an ergodic control prob- 
lem can be seen in this framework. Problems of this sort have been extensively 
considered, see for example the classical paper of Kakumanu [15) , or more recent 
work by Guo and Hernandez-Lerma [T3] . 
Consider the problem of minimizing 

J(x, U) = limsupr^oor-'^;^ [ / L{Xu-,Ut)dt 

'-J]0,T] 

where 

• U is a space of controls, which is a separable metric space, 

• U is a,U valued predictable process, 

• L : X X U ^ M. is a bounded measurable cost function, 

• E^! is the expectation under which at time t, for the path oj, X jumps 
from state to state Cj at a rate e*A'^*("^ei, for some measurable matrix 
valued function A^'^ : U — > rate matrices, and Xq = x, 

• for some 7 > 0, for all u €z 14, the matrices A" are uniformly bounded 
and strictly dominated by A with constant 7 (in the sense of Definition 
13]), for some reference rate matrix A under which X is a uniformly ergodic 
Markov chain. 

We shall write E for the expectation under which X is a uniformly ergodic 
Markov chain with rate matrix A. 
We define the Hamiltonian 

fix, z) = inf {L{x, u) + z*{A'' - A)x}. (20) 

We notice that f{-,Q) is bounded, and by the assumption that the A" are 
strictly controlled by A, it is easy to show that / is a Lipschitz function in z 
under the || • || Mt norm, and that / is strictly balanced (by testing with z equal to 
each basis vector). Therefore, the EBSDE with driver f{x,z) admits a unique 
bounded Markovian solution {Yt,Zt,X) = {v{Xt),v,X), where v is the vector 
with components v{ei). 

If this infimum is attained, then there exists (assuming the continuum hy- 
pothesis, by McShane and Warfield [T7]) a measurable function k : X x — > U 
such that 

fix, z) = Lix, Kix, z)) + z*(A'=(^'^) - A)x. 
We then have the following theorem. 

Theorem 8. In the setting described above, let (Y, Z, A) be any (possibly non- 
Markovian) bounded solution to the EBSDE (0) with driver f . Then the follow- 
ing hold. 

(i) For an arbitrary control U we have Jix, U) > X — X, and equality holds if 
and only if 

LiXt-, Ut) + Z:iA^' - A)Xt- = fiXt-, Zt) dPxdt- a.e. 
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(ii) If the infimum is attained in \20\) . then the control Ut = K(Xt,Zt) verifies 
J{x,U) = A. 

In particular, for the bounded Markovian solution to the EBSDE, in terms of 
the vector v the following hold. 

(Hi) For arbitrary controls U we have J{x, U) — X if and only if 

L{Xt-,Ut) +v*{A^* - A)Xt- =f{Xt-,v) d¥xdt-a.e. 

(iv) If the infimum is attained in i20\) . then the control Ut — K{Xt-,v) verifies 
J{x, U) = A, that is, we have an optimal feedback control. 

Proof. That A = A is a consequence of Theorem [SI and points (iii) and (iv) 
follow directly from (i) and (ii). 

To show (i), for T > define the measures, 



0,-AT] 



Z*(A'^= - A)Xs 



-Z*dM. 



As {Y, Z, A) is a solution to the EBSDE, we have 

L{Xs-,Us)ds 



A — —Eq:c,t[Yt - Yq] + —Eq^c.T 



O.T] 



0,T] 

f{X,^,Zs) - Z:{A"^ - A)Xs- - L{Xs-,Us)ds 



and hence, as / is an infimum over the controls. 



\+^E^.^t[Y^-Yt] < ^Eq 



0,T] 



L{X,^,Us)ds 



As Yq — Yt is uniformly bounded, taking a limit we see that 



A < limsup-r^^^£;Qx,T 



0,T] 



L{X,^,Us)ds =J{x,U). 



Conversely, if the infimum is attained, we have f{Xs-,Zg) — L{Xs-,Us) + 
Zs{A^' — A)Xs- for some Us = k{Xs-,Zs). Then equality holds throughout 
and we see that 



A = limsup-r_,^^£:Q 



L{Xs-,tJs)ds ^J(x,U). 



0,T1 



□ 



Remark 7. From this analysis, we also see that one could equally define the 
cost functional J{x,U) as the liminf, rather than the limsup, and the same 
conclusions would hold, with the same optimal cost A. This is simply because 
the function under consideration converges for an optimal policy, as given by 
the EBSDE solution. 



31 



5.3 Risk averse control 

The strength of the EBSDE approach is, however, not made manifest by the 
classical linear setting. Through the use of EBSDEs, one can happily consider 
nonlinear examples, in particular, when the expectation is replaced by a 
dynamically consistent nonlinear expectation, in the sense of Peng [3D] . 

For example, one could consider a form of risk averse ergodic control under 
uncertainty. In this case, we are in a similar setting to above, but the con- 
trol Us does not yield a unique rate matrix A^" , rather a family of matrices 
{A^'"'^}u]£Wu ' where Wjj^ is an index set, and the A^""^ are still all uniformly 
strictly controlled by a single rate matrix A, as in the previous section. 

To ensure dynamic consistency, we assume that Wu is such that if w and 
w' are both in Wjj^ , then there is an element w" G Wjj^ such that the ma- 
trix obtained by interchanging arbitrary columns of A"'^ and A"'"" is equal to 



In this case, we could attempt to minimise the maximum over the relevant 
measures (or some other nonlinear functional) corresponding to the value func- 
tional 



(or equivalently with a liminf). This can be directly treated by our setting, 
simply by taking the non-concave Hamiltonian 



and solving the corresponding nonlinear EBSDE. As above, one can verify that 
J{x, u) is then the A component of the solution to the EBSDE with driver 
/(m, •, •). Extending Theorem |5] to this setting is then a straightforward task. 

Remark 8. In [3J, conditions are given under which a dynamically consistent 
nonlinear expectation in a general setting can be represented by means of a 
BSDE with a balanced driver. Suppose each choice of policy yielded a nonlinear 
expectation. For each policy which makes X a homogeneous Markov process 
(in the sense that the conditional nonlinear expectation of (t>{Xt) given J^5 is a 
function of and t — s, for any measurable 4> and any s < t), the drivers of 
these BSDEs can be written in the form g„ : X x — ^ M, and assume that 
these drivers are strictly balanced. Given this representation, the Hamiltonian 
for our cost minimization problem has the generic form 



6 Conclusions 

We have seen that, for uniformly ergodic Markov chains, EBSDEs with strictly 
balanced drivers admit unique bounded Markovian solutions. The methods used 
to determine this result, while based on previous work on EBSDEs, require 
a different approach to the ergodicity of the underlying process, due to the 
presence of jumps. 

In deriving this, we have constructed a partial ordering of the rate matri- 
ces, and shown that any chain bounded below by a uniformly ergodic chain 




f{u;x,z)^ inf {L{x,u)+ sup {z* {A'''"' ~ A)x}} 



f{u; X, z) = inf {L{x, u) + gu{x, z)). 
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must also be uniformly ergodic, and that the rate coefficients can be uniformly 
bounded. In some sense, this result is similar to recent work by Galtchouk and 
Pcrgamenshchikov |13| . who study geometric ergodicity properties of general 
Markov processes, under an assumption of a uniform Lyapunov function. On 
the other hand, our method is better suited to the study of EBSDEs, where 
the perturbation of the rate matrix arises directly from the driver of the BSDE. 
Future work may allow a weakening of our assumption of uniform ergodicity to 
a form of geometric ergodicity, however we expect that this will require some 
restriction of the class of Markov chains (for example, to stochastically mono- 
tone chains). Such techniques have been used for ergodic costs from a classical 
control perspective, see for example Guo and Hernandez-Lerma [14'. Similarly 
it may be possible to weaken the assumption of strictly balanced EBSDE drivers 
to either balanced or weakly balanced. 

The applications of the theory of EBSDEs are still in development, and 
the explicit computability of solutions to these equations, in terms of solving a 
single nonlinear vector equation, is of some interest. The consequences for risk 
sensitive ergodic control, and properties of the convergence of these solutions to 
the diffusion case, remain to be explored. 
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